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Abstract. This paper is concerned with the following system of elliptic equa- 
tions 

—Au + u = F u (\x\,u, v), 
— A'« + v = — F v (\x\ , u, v), 



It is shown that if F is odd in (u, v) and satisfy some growth conditions, then 
(cS) has infinitely many both radial and nonradial solutions. The proof relies 
on the Principle of Symmetric Criticality and a generalized Fountain Theorem 
for strongly indefinite functionals. 



1. Introduction 

We study the existence and multiplicity of solutions of the noncooperative elliptic 
system 



—Am + u = F u (x, u, v), 



(S) 



-Av + v 



-F v (x,u,v), 



where F : 



u, v e H 1 ( 
is a Caratheodory function, and F u 



designates the partial 



derivative of F with respect to w. 

The natural energy associated to this problem is defined on the Hilbert space 
H 1 ^) x H 1 ^) by 



J(u, v) : = 



-f 

2 Je 



IVwI 



f (|V,| 2 +, 2 )-f 

JR N JR 



F(x, u, v). 



(1) 



This functional is strongly indefinite in the sense that its quadratic part has positive 
and negative eigenspaces both infinite-dimensional. Therefore some usual critical 
point theorems cannot be used. This is the first difficulty to overcome when inves- 
tigating the existence of solutions of (S). The second difficulty is the lack of the 
compactness of the Sobolev embeddings, since we consider the whole space. For- 
tunately when the problem has some symmetry properties, for example when it is 
invariant by a group of orthogonal transformations, it suffices to consider invariant 
functions to recover compactness. 
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There are various methods in literature dealing with symmetric strongly indefi- 
nite functionals. Among others we cite Rabinowitz [12j . Benci and Rabinowitz [5], 
Benci [J], Li [5], Bartsch and Clapp [1 and Batkam and Colin [3J. 

In this paper we assume that (S) is invariant under the actions of the groups Z2 
and 0{N), and we consider the existence of infinitely many radial and nonradial 
solutions. Inspired by Bartsch and willem [2], our approach is based on the Principle 
of Symmetric Criticality of Palais [11) and a generalized Fountain Theorem for 
strongly indefinite even functionals in [3J. 

Before we state the main results, we introduce the following conditions: 



(F x ) F e C 1 (E JV xM 2 ,M) andF(x,0,0) = VxeR^. 

(F 2 ) \F u (x,u,v)\ + \F v (x,u,v)\ < c(|u| + \v\ + \u\p~ 1 + \v\p- 1 ), with 2 <p < 2*. 

(F3) 37 > 2 such that < ^F{x,u,v) ^ uF u (x,u,v) + vF v (x, u, v), V(u,v) ^ 
(0,0). 

(F 4 ) inf {F(x, u, v) \ \{u,v)\ ^ 1, x e R N } > 0. 

(F s ) \F u (x,u,v)\ + \F v (x,u,v)\ = o(\(u,v)\), \(u,v)\ co uniformly on M N . 

(F 6 ) m.^OVieK", V(u, u) e M 2 . 

(F 7 ) F(x,u,v) = F(\x\,u,v), Vx e R ff , V(u,i>) e R 2 . 

(Fg) F(a^, — u, — v) = F(x, u, v). 
The main results are the following: 



Theorem 1. If F satisfies the assumptions (Fi) — (Fg), then (<S) has a sequence 
of radial solutions (itfc,i>fc) such that 



Theorem 2. Let N = 4 or N 6. J/ F satisfied (Fi) - (Fg), i/ien (5) has a 
sequence (yk,Zk) of nonradial solutions such that 



These theorems was first proved by Huang and Li [7] by using the Principle of 
Symmetric Criticality and the Limit Index Theory due to Li [9j. In this paper we 
propose a quite different approach which is much simpler. 

Our main results are proved in Section [3j while the require abstract materials 
for the proofs are given is section [5] 



In this section we recall the abstract results we will use in the proofs of the main 
theorems. 




k 



00. 




2. Preliminaries 
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2.1. Principle of symmetric criticality. 

Definition 3. The action of a topological group G on a normed vector space A is 
a continuous map 

G x A — > A, (g,u) h-» g ■ u 

such that 

1 • u — u, (gh) ■ u = g(h ■ u) and the map u >— > g ■ u is linear, Vg, h e G. 

A subset A of X is invariant if g ■ A = A, for every g e G. A function : A — > K 
is invariant if ip(g ■ u) = <p{u), for every u e A , g e G. 

We denote 

Fix(G) := jiiel; g ■ u = uMg e G] 
the set of invariant points. 

The following result is due to Palais [TT] (see also [T3], Theorem 1.28): 

Theorem 4 (Principle of symmetric criticality). Assume that the action of the 
topological group G on the Hilbert space X is isometric. If \p e C 1 (A, M) is invariant 
and if u is a critical point of ip restricted to Fix{G), then u is a critical point of ip. 

2.2. Generalized fountain theorem. Now we assume that A is a separable 
Hilbert space. Let Y be a closed subspace of A. On A = Y © Y 1 - we consider 
the r— topology introduced by Kryszewski and Szulkin [8]; that is the topology 
associated to the norm 

00 

|||u||| :=max(^ |(Pu, dj), \\Qu\\\), ueX, 

3=0 

where (aj)j>o is an orthonormal basis of Y, P : X — * Y and Q : X —* Z := Y 1 - are 
the orthogonal projections. 

Consider an orthonormal basis (ej)j^o of Z and define 

Y k :=Y® (e^Rfj ) , Z k : = ®f =k Rfj, 

B k := {u 6 Yk | ||u|| «£ pfc}, A fc := {ue Z k \ \\u\\ = r k } where < r fc < p fc , fc > 1. 

Definition 5. Let ip e C X (A, R) 

(1) 93 is said to satisfy the (PS) C condition (or the Palais-Smale condition at 
level c) if any sequence (u n ) cz X such that 

t /'( u n) — * c and tp'(u n ) — » 

has a convergent subsequence. 

(2) We say that <p is r— upper semicontinuous if for every Cel the set \u e 
X ; J(u) > G} is t— closed. 

(3) We say that is weakly sequentially continuous if the sequence (V<p(it n )) 
converges to Vy>(u) whenever (u n ) converges to u in A. 

The following result is due to Batkam and Colin [3]: 

Theorem 6 (Generalized fountain theorem). Let ip e C 1 (A, R) 6e an even /unc- 
tional which is r-upper semicontinuous and such that V(p is weakly sequentially 
continuous. If, for every k > ko, there exist p k > r k > such that: 



4 



C. J. BATKAM 



(Ai) ak '■= sup ip(u) =g and sup p{u) < oo. 

usY k ueY k 
\\u\\=Pk \\u\\<Pk 

(A 2 ) b k := inf ip(u) — » oo, k — » oo. 

lh"ll=T"fc 

(A3) </? satisfies the (PS) C condition, Vc > 0. 
TTien <p /las an unbounded sequence of critical values. 



3. Proof of the main results 

Throughout this section | • \ p stands for the usual L p norm, and || • || stands for 
the usual H 1 norm. We denote — > (resp. — the strong convergence (resp. the 
weak convergence). 

Definition 7. Let 1 s£ p, q < 00. On the space L P (R N ) n L 9 ^) we de/me tae 
norm 

MpAg = Hp + \v\q- 

On the space L P (R N ) + L q (R N ) we define the norm 

\u\ pvq = mi{\v\ p + \w\ q ; v e L P (R N ), w e L q (R N ), u = v + w}. 
We refer to [7] for the proof of the following lemma: 
Lemma 8. Assume that 1 < p, q, r, s < 00, H e C(R N x R 2 ) and 

\H(x,u,v)\ < C(\(u,v)\r + |(u,«)|'). 

Then, for every u,v e L P (R N ) n L 9 (R Ar ), H(-,u,v) e L P (R N ) + L q (R N ) and the 
operator 

T: (L p (R N )nL q {R N ))x(L p (R N )nL q {R N )) -> L p (R N )+L q (R N ) : (u,v) ^ H(x,u,v) 
is continuous. 

Lemma 9. J eC 1 (H 1 (R N ) x ZT^R^R) with 

(J'(u,v),((f),ip)) = (u,0)i - (v,tp)i - (0F u (x,u,?;) + (2) 
where (•, -)i denotes the usual inner product ofH 1 (R N ). 

Proof. Existence of the Gateaux derivative. Let (u,w) e if^R^) x iJ^R^). 
For every tfi, ip e H 1 (R N ) and for < \t\ < 1 we have: 

1 C t 

-[j(u+t(f),v+t(p)-J(u,v)] = (VuV<l)+u(t)-\7v\7(p-vip+-(\\7(j)\ 2 + \(f)\ 2 -\V(p\ 2 -\<p\ 2 )) 

t J R N 2 

— J - (F(x, u + tip, v + tip) — F(x, u, «)). 
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It follows from the mean value theorem that there exists As (0, 1) such that, given 
x e R N 

^-\F(x,u + t<f),v + tip) -F(x,u,v)\ «S \F u (x,u + t\(f),v + t\ip)\\(j)\ 
r I 

+ \F v (x, u + tX<f>, v + tXcp)\\cp\ 
< (\F u (x,u + t\(j),v + t\<p)\ + \F v (x,u + t\</>,v + t\<p)\)(\(j>\ + \ip\) 
s£ c(\\<f>\ + \v + tXip\ + IX^p- 1 + \v + iAp| p-1 )(|0| + \tp\) (in view of (F 2 )) 

< C (M + i^i + | V | + \< P \+2*>- i (\ur 1 + w- 1 ) +2P-\\vr i + \ipr i ))(i^i + i^i). 

The Holder inequality implies that 

C (| U | + |0| + | V | + |^|+2^- 1 (|«|^ 1 + |^r- 1 )+2^" 1 (|z;|^ 1 + |^r- 1 ))(|0| + |^l) e i 1 ^). 
It then follows from the dominated convergence theorem that 

(J'(u,v),((j>,(p)) = (u,<f)) 1 -(v,(p)i- (6F u (x,u,v) + <pF v (x,u,v)). 

Continuity of the derivative. Let (u n ,v n ) cz i/" 1 (R JV ) x H 1 ^) such that 
(u n ,v n ) — » (u, v) in H 1 (R N ) x H 1 (M. N ). By the Sobolev embedding theorem 
(u n ,v n ) — (u,v) in (L 2 (R N n L p (R N j) x (L 2 (R N n L P (M. N )). By Lemma 1 
F u (x,u n ,v n ) -» F u (x,u,v) and F v (x,u n ,v n ) -»■ F v (x,u,v) in L 2 (R N ) + L P '(R N ). 
The Holder inequality implies that 

\(j'(u n ,Vn) - J'(u,v))\ sS ||tt n - u||||0|| + ||w„ - 

) — F U (X, U, v)\2vp' \<P\2a P 

+ \F v (x ) — F v (x, U, v)\2vp' |^|2Ap- 

Hence we have 

\\J'(u n ,v n ) - J'(u,v)\\ s: \\u n - u\\\\ + \\v n - v\\ 

) - F v (x,u,v)\ 2vp ']. 

We then deduce that J'(u n , v n ) — J'(u, v) — > as n — > oo. □ 

3.1. Existence of radial solutions. We recall that the action of the group O(N) 
on i? 1 (M JV ) is defined by 

(ff • u)(x) = uig^x). 

Let HL N) (R N ) := {u e iJ^R^) | g ■ u = u, Vg e O(iV)} the set of radial functions. 
The following result is due to Strauss: 

Lemma 10 (Strauss, 1977). Let N > 2. For2 <q < 2*, the embedding H^ {N) (R N ) < 
i 9 (R JV ) is compact. 

We define 

:= Ho(N)i^- N ) x ^(jv)^") and $ := J|x- 

By (F7) J is invariant under the action of O(N). It then follows from the Principle 
of Symmetric Criticality (Lemma HJ that the critical points of $ are weak solutions 
of (S). 
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Lemma 11. 4> satisfies the Palais- Smale condition on X . That is, every sequence 
(u n , v n ) cz X such that v n )) is bounded and <&'(u n , v n ) — > 0, has a convergent 

subsequence. 

Proof. Let (u n ,v n ) c X such that 

d := sup |$(m„)| < oo and $'(w„,w„)— >0 as n^co. 

n 

We want to show that (w„, v n ) has a convergent susbsequence. 
By ([2]) and (F 6 ) we have 



( - &(u n ,v n ), (0,v n )) = \\v n \\ 2 + v n F v (x,u n ,v n ) 



Hence for n big enough we have \\v n \\ 2 ||un||- This shows that (v n ) is bounded. 
On the other hand we deduce from {T]) and ([2]) that 

$(u„,u„) - -{$'(u n ,%), K,"„)) = - ^)||un|| 2 - - ^)IM| 2 

+ [7(^^(0;, u„,u n ) + v n F v (x, u n ,v n )) - F(x,u n ,v n )] 
Jr« 

> (^-^)II^H 2 -(^-^)ll^l| 2 (m view of (F 3 )). 
Hence for n big enough we have 

(2 - ^)IM 2 - [\ ~ ^)IM 2 *s IIK,u„)|| + <*. 

Since (w n ) is bounded, we deduce that (u„) is also bounded. 

Now up to a subsequence we have {u n , v n ) —*■ (u, v) in X. By LemmallO| u n — > u 

and u n u in L P (K JV ). 

We easily deduce from (JT|) and @ that 

||u n -"|| 2 = (&(u n ,v n )—$'(u,v), (u n -u,0))+ (F u (x,u n ,v n )-F u (x,u,v))(u n -u) 

||w«-w|| 2 = -($'(u n ,v n )—$'(u,v), (Q,v n -v))- (F v (x,u n ,v n )-F v (x,u,v))(v n -v). 

Jwl n 

Clearly ($'(u n ,v n ) — $'(u, w), (w„ — u, 0)} — » as n — ► 00. 
(P2) and (F5) imply that for all e > there exists c e > such that 

|F u (x, M , V )| + < e(|«| + |v|) + c^u^ 1 + \v\p- 1 ). 

This implies that 

\(F u (x,u n ,v n ) - F u (x,u,v))(u n - u)\ i< e(|u„| + |« n | + |w| + |w|)(|u„| + \u\) 

+ C s {\ Un \V- 1 + \u\P- 1 + \Vn\P- 1 + ^r 1 )!^ - 4 

Since the sequence (u n ,v n ) is bounded in X, we obtain by using the Holder in- 
equality 

\(F u (x,u n ,v n ) - F u (x,u,v))(u n - u)\ C(e + c e \u n - u\ p ), 
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where C is constant independent of e and n. It is then easy to see that u n — » u as 

n — > oo. 

By the same way we show that v n — * v as n — > oo. □ 
Lemma 12. V$ «s weakly sequentially continuous. 

Proof. Let («„,«„) c X such that (u n ,v n ) — (it, u) in X. (|2|) and the Holder 
inequality imply, for any cj),ip e X 

\(j'{u n ,v n ) - J'(u,v), (0, ^ (V(u n - u)V^+ (u n — 

Jr« 1 

(V(t> n - «)VyJ + (v n - v)lf) + \F u (x ) - F u (x,U,v) L ,|0|2 



+ 

+ 1*1 ( 

) — F„(ar, u, w)| 2vp ,|y|2A P - 

It is clear that 



0. 



(V(it„ — u)V<t> + (u n — u)(j)j — > and (V(u n — v)V<p + (u n — u)</j) 

By Lemma [TOl u„ — ► u and u n — » u in L 2 (]R Ar ) n ^(l*). We then deduce from 
Lemma[8]that F u (x, u n ,v n ) — F u (x,u,v) —> and F v (x,u n ,v n ) — F v (x,u,v) — > 
in L 2 (IR Ar ) + ^'(R"). 

It then follows that J'(u n ,v n ) J'(u,v). □ 



We define 



y := {0} x Hi(R N ), Z := HL N) (R N ) x {0}, 



'0(AT)V m J; ^ ■— 11 0(N) 

and we consider the r— topology on X = Y © Z. 
Lemma 13. $ is r— upper semicontinuous. 

Proof. Let w„) cz X and Cel such that (u n , v n ) —* (it, v) in X and J(u n , v n ) ^ 
C. By the definition of t, m„ — > u in if^^-^R^). 

1 If 

J(u n ,V n ) ^ C -\\v n \\ 2 + C ^ -\\u n \\ 2 - F(x,U n ,V n ). 

Z Z J k jv 

Since F ^ we deduce that 

^Kll 2 + C< i|| Wn || 2 . 

Since (u„) is bounded, we easily deduce that (v n ) is also bounded. 
Now we may suppose, up to a subsequence that 

(u n ,v n ) — (u,v) in X, 

u n (x) — > it(x), — » i>(x) a.e. in 1*, 

^(a;, u n (x), v n (x)) — » F(x,u(x),v(x)) a.e. in M. N . 

It follows from Fatou's lemma and the weak lower semicontinuity of the norm || • || 
that - J(u, v) < -C. □ 

Proof of Theorem^ Let (ej)j^o be an orthonormal basis of H^^ N ^(M. N ). Set 
for k > 2, 

y fe = y © ( Rej- x {0}) and Z fe = Rej x {0}. 

3=0 j=k 
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Let (u, v) e By (F 3 ) and (F4), for all S > there exists Ci = Ci(<5) > such 
that F(x,u,v) > Ci|(u,w)| 7 — S\(u,v)\ 2 . This implies that 

$(u,v) < i||u|| 2 - ^H| 2 + 5(\u\ 2 + \v\ 2 ) - Ci(\up + M 7 ) 

^IMI 2 -^MI^(M 2 + M 2 )-GiM 7 - 

k 

Since all norms are equivalent on (J) Re^, we obtain 

*(«,«) < (i + 5) || U || 2 - (5- i)||tf - C 2 Ci||u|| 7 
where C2 > is a constant. By choosing <5 < 4, we obtain 

Jiiu|| 2 -i|| W || 2 -ci| M ir. 

Hence — * -co as u)|| — * +00, and assumption (A\) of Theorem [6] is then 

satisfied for pk sufficiently large. 

Now let (u, 0) e Zk- Let e > 0, by (F2) and F3 there exists C E such that 
F(x,u, 0) s£ e|u| 2 + C e |u| p , which implies 

*(«,o) > l\\uf - £ | U | 2 - c £ K > (i - e ) mi 2 - c e \u\ p - 

By choosing e < | we obtain 

a>Ko)>i|| M |i 2 -q u r>i|| M |i 2 -^iiur, 

where 

:= sup |w| p . 

|| || = 1 

Let 

r fc := {2pCfa)^. 
Then for any (u, 0) e such that = we obtain 

$(u, 0) > ( j - i)r| qo, as fc — go, 

since /3k —* as —* 00 ( see [T3] Lemma 3.8). The assumption (^2) of Theorem 
[6] is satisfied. 

By (F$) $ is even, and by Lemma [TT1 the assumption (A3) of Theorem [6] is satisfied. 
We then conclude, in view of Lemmas El [12] and [13] by applying Theorem |U □ 

3.2. Existence of nonradial solutions. Let N = 4 or N ^ 6. let 2 s? m < N/2 

an integer different from (AT — f)/2. We recall that the action of the group G : = 
0(m) x 0(m) x 0(N - 2m) on if 1 (R Ar ) is defined by (5 • u)(x) = u(5 _1 x). Let 

H^,(R N ) := {w e iJ^R^) ;g-u = uVgs G). 

Lemma 14 (P. L. Lions, [TO]). For 2 < q < 2* , the following embedding Hq(M. n ) «-» 
Z/ 9 (M JV ) is compact. 
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Denote i the involution denned on R N = R m x R m x R N - 2m by 

i(xi,X2,x 3 ) := (x 2 ,xt,x 3 ) 
The action of H := {i<i K N,i} on Hq(M. ) is defined by 

v /y ' [ — a;) u h = l. 

By this construction, is the only radial function in 

h g.h( rN ) { u e h g( rN ) ;h-u = u,\/heH}. 

Proof of Theorem^ We define 

X := Hq H (R n ) x Hq H (R n ) and*:=J| x . 

By (Fy) and the Principe of Symmetric Criticality (Theorem [4]), the critical points 

of \I/ are also critical points of J. 

Consider the r— topology on X = Y © Z, where 

Y := {0} x Hq H (R n ) and Z := Hq M (R n ) x {0}. 

The rest of the proof follows the lines of the proof of Theorem [T] above. □ 
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